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Abstract — It is anticipated that an uncoordinated operation 
of individual liome energy management (HEM) systems in a 
neighborhood would have a rebound effect on the aggregate 
demand profile. To address this issue, this paper proposes a 
coordinated home energy management (CoHEM) architecture 
in which distributed HEM units collaborate with each other 
in order to keep the demand and supply balanced in their 
neighborhood. Assuming the energy requests by customers are 
random in time, we formulate the proposed CoHEM design as 
a multi-stage stochastic optimization problem. We propose novel 
models to describe the deferrable appliance load (e.g.. Plug-in 
(Hybrid) Electric Vehicles (PHEV)), and apply approximation 
and decomposition techniques to handle the considered design 
problem in a decentralized fashion. The developed decentralized 
CoHEM algorithm allow the customers to locally compute their 
scheduling solutions using domestic user information and with 
message exchange between their neighbors only. Extensive simu- 
lation results demonstrate that the proposed CoHEM architecture 
can effectively improve real-time power balancing. Extension to 
joint power procurement and real-time CoHEM scheduling is 
also presented. 

I. Introduction 

Demand side management (DSM) techniques have been 
employed to make the inelastic demand for electricity flex- 
ible in order to achieve the goal of integrating intermittent 
renewable energy resources in the power grid [1], [2]. Existing 
DSM techniques can be divided into two categories- Direct 
load control (DLC) and dynamic pricing. DLC mechanisms 
[3], [4] allow electric utilities to conduct centralized demand 
management on certain interruptible appliances, e.g., turning 
off air conditioning systems for short periods of time, to 
maintain demand and supply balance during peak hours. On 
the contrary, dynamic pricing schemes [5], [6] distribute a 
control signal to the customers that reflects the congestion of 
the grid, counting on the assumption that individual customers 
will modify their demands accordingly. Since a manual control 
in response to ever-changing price signals is infeasible, dy- 
namic pricing solutions require intelligent energy management 
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software to manage the demands of residential and commer- 
cial customers. Consequently, there is an extensive literature 
emerging on home energy management (HEM) systems, e.g., 
[7]-[10], dedicated to finding optimal scheduling algorithms 
for the household appliances, based on the price signals, 
appliance load profiles, job deadlines, etc., with the goal of 
minimizing the electricity bill. 

As observed in [11], if all the customers in a neighborhood 
are given the same dynamic price, the HEM systems that are 
individually operated by each customer will simultaneously 
schedule the load to the low-price period, and, consequently, 
a new "rebound" peak may occur Another concern is that 
dynamic prices may render the demand more volatile and less 
predictable, causing serious stability and reliability issues for 
the grid if not handled properly [12]. In this paper, we aim 
to blur the boundaries between DLC and dynamic pricing 
strategies by proposing an architecture through which the 
HEM units inside the territory of an aggregator/retailer can 
cooperate with each other to keep wholesale demand of the 
retailer balanced with the available generation supply (which 
might be the day-ahead/hour-ahead bid plus locally generated 
renewable resources). 

Related works: Several existing works have studied 
neighborhood-wise collaborative energy management, though 
different models and optimization goals are considered. For 
example, in [11] a heuristic algorithm is proposed for schedul- 
ing the load of customers in a neighborhood to meet a 
maximum power profile specified by the retailer In [13]-[15], 
distributed energy management algorithms, based on game- 
theoretic approaches, were proposed to minimize the cost of 
the retailer or the peak-to-average ratio of the aggregate load. 
The works in [16], [17] and [18] proposed distributed energy 
management algorithms that maximize the social welfare by 
minimizing both the costs of retailer and customers. The work 
in [19] considered simultaneous procurement of power in the 
day-ahead market and real-time load scheduling to maximize 
the social welfare. However, most of the literature cited above 
does not account for the customers' probabilistic behavior in 
the usage of appliances, which is very important for making 
decisions that meet the current need of the customers as well as 
for predicting what else the customer may want in the future. 
Moreover, a common assumption in these papers is that the 
HEM system can adjust the appliance power consumptions, 
which may not be valid for some non-interruptible appliances. 
In [20], [21], a neighborhood-level load scheduling architec- 
ture is proposed by modulating and scheduling the demand 
of certain deferrable appliances. This architecture, however, is 
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implemented in a centralized fashion and cannot guarantee the 
deadline constraints for customers. 

Contributions: In this paper, we propose a coordinated 
HEM (CoHEM) architecture where the HEM units in a 
neighborhood collaborate to minimize the cost of their ag- 
gregator/retailer in the real-time balancing market. Similar 
to previous works [8]-[10], [15], we assume that the retailer 
broadcasts dynamic pricing information to the residences, and 
that the associated HEM systems optimize the scheduling of 
their local appliances. Unlike [8]-[10], and similar to [13], 
[16]-[19], the CoHEM customers will not be selfish. One of 
the key differences of our model compared to [11], [13]-[19] is 
in the choice of the network utility and in the effective pricing 
and service policy applied to the HEM users. Specifically, 
in our model, the HEMs, by cooperating, do not pay more 
than their selfishly optimized cost and do not experience lower 
quality of service. The network utility of the CoHEM is chosen 
to be equal to the cost of deviating in real time from the bulk 
power purchase of the retailer in the day-ahead and hour-ahead 
markets, computed using the locational marginal prices [22]. 

Another major difference compared to [13], [16]-[19], but 
similar to [10], [11], [15], is that our work focuses on appli- 
ances that are flexible in deferring their operating times, e.g., 
PHEV, washing machine, dish washer and tumble dryer etc. 
(which are usually non-interruptible in power consumption). 
Moreover, in order to be foresighted, we take into account 
the customers' probabilistic behavior by assuming that the 
customers randomly submit requests to their HEM to use 
an appliance. Given the statistical information of customers 
and quality of service constraints (i.e., scheduling deadline 
constraints), we formulate the proposed CoHEM design prob- 
lem as a multi-stage stochastic optimization problem [23], 
[24], aiming at minimizing the expected real-time power 
unbalancing cost of the retailer The stochastic formulation 
can provide optimal control policies which can be used for 
real-time appliance scheduling by exploiting both customers' 
statistical and real-time request information. 

To this end, we first present a Markov decision process 
(MDP) formulation that can efficiently solve the selfish HEM 
design problem. Subsequently, we show that the MDP method 
can be used to develop a decentralized scheduling algorithm 
for efficiently handling the proposed CoHEM design prob- 
lem. It is known that a centralized computation requires full 
knowledge of all the statistical and real-time information 
of the customers, and moreover, the required computational 
complexity increases with the number of residences and the 
number of controllable appliances. In view of this, a distributed 
implementation algorithm, that can decompose the original 
problem into parallel subproblems with smaller problem sizes, 
is of great interest. Such a distributed algorithm can be 
deployed in the neighborhood in a fully decentralized fashion 
where each of the residences computes its scheduling solution 
locally using domestic information and by communicating 
with its neighbors only. Since no explicit information about 
customers' electricity usage is exchanged and submitted to the 
retailer, this decentralized method also preserves customers' 
privacy with respect to their appliance usage. Extensive simu- 
lation results will be presented to demonstrate the effectiveness 



of the proposed CoHEM architecture and the decentralized 
scheduling algorithm. 

Synopsis: In Section II, we first present the load models 
of deferrable appliances, and the individual (selfish) HEM 
design problem. Secondly, we present the proposed CoHEM 
architecture and the associated CoHEM design formulation. 
In Section III, an MDP method for solving the selfish HEM 
design problem is proposed. Then, based on this MDP method 
and decomposition techniques, we propose a decentralized 
CoHEM algorithm. Two extensions of the proposed CoHEM 
design are also discussed in the last subsection. Extensive 
simulation results are presented in Section IV. Finally, the 
conclusions and future directions are included in Section V. 

II. Appliance Load Model and Problem Statement 

In the first two subsections, we present the load model 
of deferrable appliances and the individual HEM problem 
formulation. The proposed CoHEM architecture is presented 
in the third subsection. 

A. Multi-Mode Deferrable Appliance Load Model 

We consider the case where there are N deferrable ap- 
pliances in each residence. The appliances are assumed to 
have known power consumption profiles, and once are turned 
on, their operation cannot be interrupted, e.g., PHEV, dish 
washer, tumble dryer etc. The HEM system in the house is 
allowed to defer their schedules within the deadlines specified 
by the customers. Specifically, given a request submitted by the 
customer, the HEM unit has to decide to turn on the appliance 
immediately or defer the task by waiting in a queue. The 
decision process is repeated until the HEM system chooses 
to activate the appliance or until the maximum delay time is 
reached. 

In the paper, we assume that the appliance may have multi- 
ple operation modes, each with a different power consumption 
profile, and they are decided by the customer For example, 
the washing machine may have one mode for colored clothes 
and one mode for white clothes. We assume that appliance 
i has Ali modes, and each mode specifies a (discrete-time) 
power load profile gi,m{t), t = 1, ■ • • , Gi.m, where Gi^m. > 
is the maximum job length of gi^m{t)- The times at which 
the customer submits a request for an appliance are random. 
In particular, at each time t, appliance i will be requested 
by the customer with probability Pi{t) £ [0,1], and, once 
it is requested, it is with probability ji^m{t) that mode m 
wiU be chosen {J2m=i^i,m{t) = 1), where t = 1,...,T, 
with T > denoting the maximum look-ahead time horizon. 
Information about Pi{t) and ■yi^m{t) can be estimated through 
the usage history of customers; see, e.g., [25], [26] for related 
papers. Suppose that requests for turning on appliance i arrive 
at times ii.2, • • ■ G {i, ■ ■ ■ ,T}. Moreover, let 9i{ti^k) G 
{!,..., Mi] denote the operation mode chosen at time ti,k- 
Then, without scheduling, the power load due to appliance i 
would be 

oo 

L^{t)^Y.9^,e^iu,,){i-U,k) Vi. (I) 
fe=l 
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The requested appliance tasks may be queued and scheduled 
to operate later Let s^.i, Si,2, ■■■ 6 {1,...,T}, be the 
scheduled times determined by the HEM system for turning 
on appliance i, where ^ > ^ for all k. Then, the scheduled 
power load of appliance i is given by 

oo 

Di{t) = ^9i,eiiu,k)it - ^i.k) Vt. (2) 
fe=i 

Taking into account the power load of the uncontrollable 
appliances, denoted by U{t), the aggregate power load of a 
residence can be expressed as 

N 

itotal(t) = U{t) + ^ A(t), t = 1, . . . , T. (3) 

1=1 

B. HEM Design Problem 

Let 7r(t), t ~ 1, . . . ,T, be the dynamic electricity prices 
given by the retailer The HEM targets to schedule the con- 
trollable appliances such that the expected total electricity cost 
of the customer, i.e., 

T 

^E{^(i)Ltotai(i)} (4) 

t=l 

is minimized, where E{-} denotes the expectation operator 
The scheduling task is usually subject to constraints that reflect 
the customer's degree of comfort. In this work, we assume 
that the customer will preassign a maximum tolerable delay 
for each appliance, and that the HEM system must not exceed 
the specified delay. In particular, we let Ci_,„ > denote the 
maximum delay for mode m of appliance i. Then the operating 
times of appliance i have to satisfy 

ti,k < Si,k < ti,k + (,i,ei{u,k) ^k. (5) 

Mathematically, the HEM design problem can be formulated 
as the following multi-stage stochastic optimization problem 

oo 

s.t. D,{t) = E 9ifii{u,k) - (6b) 
fe=i 

Si,k < max{T - Gj.(,,(i.^),ii,fc} Vi, fc, (6d) 

where, in (6a), the expectation is with respect to the random 
arrival times {ti ^}, random operation modes {Oi{ti k)} and 
the control variables {si./t}, which are all based on the statis- 
tical usage information of the customer The constraints in (6c) 
and (6d), however, specify the real-time scheduling constraints. 
Specifically, (6c) implies that the customer won't wait longer 
than the specified delay in real time; while (6d) implies 
that all the requested tasks have to be finished before time T if 
they arrive before time T — Gi,g.(^ti t); otherwise, they should 
be activated for service right after their arrivals ' . 

'Note that if (6d) is not imposed and if C,i,Oi{ti j.) ^ G^ g^it^ the 
scheduler would not turn on the appliance until the end of the horizon T, 
since it contributes no cost to the objective function in (6a). 



An important aspect for the multi-stage stochastic formula- 
tion (6) is that, according to the stochastic optimization theory 
and dynamic programming techniques [23], [24], we can find 
a so called optimal control policy^ for {si^k} that, based on 
the customer's specific real-time requests, can schedule the 
appliances in real time satisfying the real-time scheduling 
constraints (6c) and (6d) while minimizing the expected cost 
in (6a). For the HEM design problem in (6), however, it 
is intrinsically too difficult to obtain such optimal policy in 
its current form. In Section III-A, we will show that an 
optimal control policy of (6) can be efficiently obtained by 
reformulating (6) as a Markov decision process [23]. 

C. Proposed CoHEM Design 

Dynamic prices {7r(t)}^]^ are designed by the retailer so 
that the customers would move their load to use cheaper 
electricity and mitigate congestion in the grid. As mentioned 
in the introduction, due to the rebound peaks, the aggregate 
power load from multiple HEM-based customers may not 
necessarily follow the bulk power purchase of the retailer, and 
the resultant real-time power imbalance will increase the cost 
of the retailer in the wholesale real-time balancing market. 
Specifically, in addition to purchasing electricity in the day- 
ahead and hour-ahead markets, the retailer has to purchase 
additional amount of energy in the real-time balancing market 
or pay the grid for absorbing the excessive energy that cannot 
be consumed. The profit of the retailer is roughly 

Profit = Be- CostRT - CostDHA (7) 

where represents the total money paid by the customers for 
their electricity usage, CostjjHA denotes the cost already paid 
by the retailer in the day-ahead and hour-ahead markets, and 
Cost/fT denotes the cost in the real-time market. As discussed, 
the selfish HEM systems will potentially increase Cost jit- 
Hence it is desirable to coordinate the customers to reduce 

CoStijT"'- 

Let TTp{t) be the price for buying energy from the real- 
time market and tTs (t) be the price for absorbing extra energy 
(if TTs{t) < then it implies that the retailer may sell 
back the extra energy). Let P{t) be the bulk power already 
purchased in the day-ahead and hour-ahead markets {P{t) may 
also contain powers generated by local renewable sources, 
if applicable). Moreover, assume that there are a total of 
H residences/customers, each contributing the power load 
-^totai(^) [^^^ (^)] '■^ '■^^ S^^'^- The total real-time market cost 
of the retailer Cost^jT is given by^ 

-By stochastic optimization [23], [24], a control policy is a function of the 
problem states and is hke a table that lists all the corresponding actions that the 
controller should follow for all possible states of the problem. Once a control 
policy is obtained (e.g., by the dynamic programming techniques [24]), the 
controller can control the appliance in real time by applying the action that 
con'esponds to the specific real-time state information of the appliance. A 
control policy is said to be optimal if it minimizes the expected cost function 
of the considered problem. 

'The reduction in CostjjT implies that the retailer can make more profits, 
and in turn the retailer may consider reducing the electricity price of the 
customers. Hence the CoHEM program is potentially economically beneficial 
to the cooperative customers from a long terni perspective. 

''The real-time cost can be extended to general convex functions; see Section 
III-C. 
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CostflT = ^ 



(8) 



where = max{a;,0}. As incentives for the customers 

to participate in the proposed CoHEM architecture, we pro- 
pose that 1) the retailer will charge the customers the same 
amount of money as that optimized by their individual (selfish) 
HEM system [i.e., (6)]; 2) the CoHEM will maintain the 
same scheduling deadline constraints specified by each of the 
customers. In summary, the CoHEM customers would neither 
have any financial loss nor would lose any degree of comfort. 
The retailer, which presumably will have infrastructure cost 
to cover, seeks to make a profit by minimizing Cost ht under 
the strict deadline constraints. Hence, the actual degree of 
flexibility of the community customers affects the retailer's 
profit. 

The proposed CoHEM design problem can be formulated 

as 



mm 

V i,h 



\h = l 



i = l 

constraints in (6b), (6c), (6d) Vi, fc, h, 



(9a) 

(9b) 
(9c) 



where we use superscript (/i) to denote the /ith resi- 
dence/customer; for example, t^l"^ and s\}. represent the 
request arrival and task operating times of appliance i in the 
/ith residence. Similar to the selfish HEM design problem 
(6), for the CoHEM problem (9), we aim to find an optimal 
control policy for {sj').'} for all fc,i and /i, in order to 
schedule the appliances satisfying the real-time needs of the 
customers while minimizing the expected real-time market 
cost in (9a). Note that problem (9) is subject to the same 
scheduling constraints as (6) for each residence, meaning 
that the degrees of comfort of customers are preserved in 
the CoHEM architecture. However, different from (6) which 
is solved by each individual residence independently, the 
CoHEM problem (9) and its associated scheduling policy have 
to be jointly optimized across all the residences in order to 
minimize the real-time market cost of the retailer 



III. Proposed Solutions for HEM and CoHEM 

In this section, we present the detailed methods for solving 
the HEM design problem in (6) and handling the proposed 
CoHEM design problem in (9). In Section III-A, we show 
that the HEM design problem (6) can be efficiently solved 
by reformulating it as a Markov decision process (MDP). 
Regarding the CoHEM design problem (9), as discussed in 



the introduction section, our interest lies in a decentralized 
scheduling algorithm. To this end, we present in Section III-B 
a suboptimal but efficient decentralized algorithm for handling 
(9), based on proper problem approximation and Lagrange 
dual optimization techniques [27]. It will be shown that the 
MDP method presented in Section III-A can be conveniently 
employed by the proposed decentralized algorithm for com- 
puting the load scheduling policy of each individual residence. 
Finally, two interesting extensions of the proposed CoHEM 
architecture are presented in in Section III-C. 

A. Solving HEM Problem (6) by MDP 

In the subsection, we show how (6) can be efficiently solved 
by an MDP formulation. Note that, for the HEM design 
problem (6), we can focus on optimizing one appliance, say 
appliance i, as follows 



mm 

Si, 1, Si, 2, 




(10a) 



\k=l 



S.t. < s^^k < U,k + C:,ei(t,,fc) Vfc, (10b) 
Si,k < max{T - Gi^e.(4.^), ii,fc} Vfc. (10c) 

According to the appliance model described in Section II-A, 
once the customer requests to activate an appliance, the appli- 
ance is either operated immediately or it waits in a queue to 
be turned on later by the scheduler We use the three variables 
S,it) e {0, 1, . . . , M,}, W^{t) e {0, . . . , max,„=i,...3/ G,,™} 
and Qi{t) £ {0, 1, . . . , maxm=i^...^M Ci.m} to denote the 
operation mode, the remaining job length and the remaining 
maximum delay time for appliance i at time t. Moreover, we 
use Ui{t) e {0, 1} as a control variable that switches on and 
off the appliance at time t. 

When all Si{t),Wi{t) and Qi{t) are zero (i.e., 
{S^{t),Wi{t),Qi{t)) = (0,0,0)), the appliance is idle, 
and consequently Ui{t) ~ 0. Then it is with probability 
Pi{t + l)7i,m(t + 1) that the customer will request to activate 
appliance i at time t + 1 and operate it in mode m. If appliance 
i is called by the customer to operate in mode m, then we have 
iS,{t + 1), W,{t + 1), g,(i + 1)) = (m, G,,mX^,m), and the 
HEM system has to decide whether to activate the appliance 
or to queue the task by deciding Ui{t+1) = 1 or u,; (i + 1 ) =0. 
If the controller chooses to activate the appliance, then the 
appliance has to work for Gi m time slots consecutively, 
following its load profile gi,m(l), • ■ • , g^mlGj,™) (i.e., 
Ui{t + 1) = Ui{t + 2) = • ■ • = u,{t + G^^m) = !)■ After 
that, it is with probabiHty + d + l)^t.rn'{t + + 1) 
that the appliance is called (for mode m') and goes back 
to state {m' ,Gi^m' ,Q,m')', otherwise (with probability 
1 - Pt{t + Gi + 1)) it goes back to state (0, 0, 0) (idle). 

If, at state {m,Gi^rmCi.m), the controller chooses to delay 
the appliance {ui(t + l) = 0), then the state moves to (S'i(t + 
2), W^{t + 2),Q,{t + 2)) = (m, G,,™, C»,m - 1) in the next 
time slot, and the HEM system needs to decide if Ui{t + 2) 
should be one or zero. The decision process is repeated until 
the HEM system chooses to turn on the appliance or until 
the maximum delay time is reached. In both cases, the state 
moves to (m, Gi^,„ — 1,0). An example of a 2-mode appliance 
is illustrated in Figure 1. 



5 




Fig. 1: Illustration of the Markov process for modeling a 2-mode deferrable appliance with Mi = 2, Gi^i = 4, Cj.i = 3, 
Gi.2 = 2, Cj,2 = 1, Pi.i{t) = Pli and Pi;2{t) = Pl2- The triplets in the circles stand for the state (S*, Q) of the appliance. 



Let Xi{t) = {Si{t),Wi{t),Qi{t)) be a state vector, and Xi 
be the set that contains all possible X, (t)'s. Using the above 
Markov process model, the HEM design problem (10) can be 
rewritten as the following MDP: 



mm 
t=l,....l 



T 

E 

t=i 



E{TT{t) (u,;(i)5.,s.(t)(G,,s.(t) - W,{t) + 1))} 



(11a) 



X,{t + l) = 



(lib) 
(11c) 



s.t. Hi^tiXi{t)) = Ui{t), 

w^{x,{t),u,{t),e,{t + l)) 
_Q,{x,{t),u,{t),e,{t + i)) 

yt = i,...,T, 

Ai,;,t (5,; (t) , G,,„„ g, (0) = 1 V t > T - G,;, (t 

where 9i{t + 1) G {0, 1, ... , A/,} is a random variable with 

Prob(6l,(t + l) =m) =K(t + l)7».m(^ + l) and Prob(6l,(t + 
1) = 0) = 1 —piit + 1), indicating the operation mode of the 
appliance at time t-\-l provided that it is requested. Moreover, 
Si{-), Wi(-) and Qi{-) are the state transition functions given 
by (see Figure 1 as an example) 

e,{t + i) if (0 = 0, 

Si{t) otherwise, 



S,{X,{t),e,{t + l)) 



Wi{x,{t),u,{t),e,{t + i)) 

W,{t) - 1 if S,{t) = TO,2 < W,{t) < G,,^,u,{t) = 1, 
W,it) ifW^{t)>l,u,{t)=0, 
G^.ra if < W^{t) < 1 and 6'i(t + 1) = m, 
if < W.,{t) < 1 and 0,{t + 1) = 0, 

Q,{X,{t),iH{t),9,{t + l)) 

[ Q,{t) - 1 if S,{i) = m, W,{t) = G,.,n,u,{t) = 0, 

Ci,m if < w,it) < 1, e,it + 1) = 772, 

if S,{t) = 771,2 < W,it) < G^,ra,U,{t) = 1, 

or if Wi{t) = 1 and 9i{t + 1) = 0, 

and (11c) is due to (10c), which enforces the requested 
appliance to turn on if its task is still queued for t > T — Gi^m- 
In (11), the optimal control policy for Ui{t), denoted by 
fii^t{Xi{t)), is a function of Xi{t), for all Xi{t) G Xi and for 
alH = 1, . . . , T. Once the control policy lJ.i,t{-), t ~ 1, . . . ,T, 



= < 



is obtained, the controller can schedule the appliance in real 
time, following the policy given the real-time state status 
Xi{t) of the appliance. 

By dynamic programming (DP) [24], ^i^t{Xi{t)) can be 
obtained by considering the following backward recursive 
equations 

Jt{X,{t))^ min 7r(t)u,(t)5,,™(G^.„-W,(t) + l)+ 

"i(*)6{04} 

Ee.(t+i){Jt+i(X,(t + l))} 
s.t. Si(t) ~ m, 

constraints in (1 lb), (1 Ic), (12) 

for all possible Xi{t), t = 1,...,T — 1. Specifically, 
l^ii^t{Xi{t)) is given by the optimal Ui{t) of (12). The value of 
Jt{Xi{t)) is known as the cost-to-go function [24]. One can 
show that (12) can be classified into the following four cases 
(see Figure 1 as an example): 

Case 1 {S,{t) ^ Q^V^{t) = 0, Q,{t) = 0): M*,* (0,0,0) = 0, 
and 

MO, 0, 0) =(1 - p,it + 1)) Jt+i(o, 0, 0)+ 

Mi 

Pi{t+l) ^ %,mJt+lim,Gi^„i,Ci.m)- (13) 
m— 1 

Case 2 (S,{t) = 771, W^jt) = G,,m, 1 < Q^it) < 0,m): For 
t > T - G^^rn., fJ-i,t{m, Gi^rn, Qi{t)) = 1 according to 
(11c); otherwise Hi.tim, Gi^m, Qi{t)) is given by the optimal 
solution of the following problem 

Jt{ni,Gi^rn,Qi{t)) 

= min 7T{t)ui(t)gi_rnil) + 
'"i(t)e{o,i} 

Jt+iim,W,{t + l),Q^it + l)) 

s.t. W,{t+1) =G,^rn-U,{t), 

'0, ifw,(0 = l, 
g,(i)-l, ifu,;(t)=G. 

Case 3 {S,{t) = m, 2 < W,{t) < G,^rn., Qt{t) = 0): 



fi^,t{m,Wt(t),0) = 1, and 

Jt{m, W,{t),0) =^(t)5.,r„(G,,™ - W^{t) + 1)+ 

Jt+i{m,W,{t)-l,0). 
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Case 4 (Si{t) = m, W^(t) = 1, Q,{t) = 0): ^i,t(m, 1,0) = 
1, and 

Jt(m, 1,0) =TT{t)g,^^{G„-„) + (1 + l))Jf+i(0,0,0) 

Mi 

+ Pi{t+1)^ Ji^mJi+lim, Gi^raXi,ra)- 
m — 1 

It can be seen that, only for Case 2 (e.g., the gray states 
shown in Figure 1), one is required to compute the optimal 
control policy /ii t(Xi(t)). Therefore, at each stage of the 
backward computation, one needs only compute and store 
^i^t{Xi{t)) for X^mLi Ci,m possible states. Since both Mi and 
Ci^m are usually small numbers, the backward computation 
can be implemented very efficiently. By applying the MDP 
approach to each of the appliances, we obtain the optimal 
appliance control policy for the HEM design problem (6). 

Remark 1 In the Markov model presented above, the appli- 
ance has multiple modes which are decided by the customer 
Sometimes, each mode may have multiple power consumption 
alternates that all can fulfill the same task. In that case, the 
controller has a further degree of freedom to choose the one 
that is best out of all the available consumption profiles in 
minimizing the customer's cost. This appliance model can be 
easily extended from the current Markov model described in 
this subsection. Specifically, suppose that, for an appliance, 
each mode has 2 alternate power profiles to choose from, e.g, 
9i,m.i{t) and gi^m,2{t)- The control variable u,;(i) then has 
three possible values - means WAIT (or remain IDLE if not 
requested), 1 means ON following profile 1, and 2 means ON 
following profile 2. The state vector for this case is given by 
X,it) = {S,it),R,it),W^{t),Q,{t)) where R,{t) G {0,1,2}. 
An example for Mi ~ 1 (single mode), Gi^i = 3, Ci.i — 2 
and Pi^i{t) — p \s illustrated in Fig. 2. The associated optimal 
control policy for the HEM design problem (6) can be derived 
in a similar fashion as described in this subsection. 

B. Decentralized Stochastic Optimization for CoHEM 

While the Markov process model presented in the previous 
subsection can also be used for the CoHEM problem (9), 
the resultant MDP problem would involve a large number 
of states (which increases exponentially with H and N) and 
thus the solution quickly becomes computationally unafford- 
able. A simple approximate DP approach to overcoming the 
curse of dimensionality issue is the model predictive certainty 
equivalent control (CEC) method [24]. In this method, one 
searches for the optimal control in a forward manner and 
apply the control at each time that would be optimal if the 



uncertain quantities, i.e., t^'j^? and 9l"''{ti_k), were fixed at 
some typical values, e.g., the mean values. The advantage 
of this method is that at each time t, one only deals with 
a deterministic optimization problem; the associated solution 
can be directly applied to control the appliances in real time. 
In [28], we have proposed a decentralized CoHEM algorithm 
(i.e., [28, Algorithm 1]) based on the model predictive CEC 
method. While the algorithm in [28] can be implemented in a 
decentralized fashion (i.e., [28, Algorithm 2]), the controller 
has to repeat the optimization until the end of the horizon. 



u = l 



IDLE 






^ ) 0, 3, 2) ^ (1,0, 3, 1) > (1, 0, 3, 0) 

Requested 
■ Mode 1 





Fig. 2: Illustration of the Markov process for modeUng a 
single-mode deferrable appliance {Mi — 1) with two alternate 
power consumption profiles, and Gi,i ~ 3, Q.i — 2 and 
Pi.i{t) = P- The quadruples in the circles stand for the state 
{S, R, W, Q) of the appliance; see Remark 1. 

from time 1 to time T — 1. Moreover, since the decentralized 
optimization algorithm involves multiple iterations, each of 
which requires the customers to exchange messages with their 
neighbors in real time. As a result, the model predictive CEC 
algorithm in [28] may cause a considerable communication 
overhead in the CoHEM network. In this subsection, we 
present a decentralized stochastic optimization method which 
can greatly alleviate the communication overhead. 

Unlike the model predictive CEC, the idea here is to 
maintain the stochastic nature of the CoHEM problem (9), 
using a suboptimal formulation as follows 

T , / H N \ + 

,1 '".,2 >■■■ t=l ^ \ h=l i=l / 

+ '^pWfEElE[i?f)(t)]-P(t)') I (14a) 

\h=l i=l ) ' 



V i,h 



s.t. Dl^^(t)=Y^gl'^it-sl^)y^,t,h, 



(14b) 



fe=i 



tS<.S<tS+<i^,<^,Vz,fc,/., (14c) 
s^! <^^{T~G^,,,^,^,^,tl^}y^,k,h, (14d) 

where P{t) = P{t) - Y.h=i U^''\t). It can be seen that the 
objective function (14a) is instead the deviation between the 
expected aggregate load and the power supply P{t), which is 
a lower bound of that in (9a). Note that (14) is still a (multi- 
stage) stochastic optimization problem. 
Define 



H N 



Z(t) 



^^E[i?f'(t)]-P(t) 



(15) 



\h=l i=l 
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and note that 



= ^w- EE^[A*"w]-m ■ (16) 

\h=l j=l / 

Substituting the above equations into (14) gives rise to 

T 

min S2iTTsit)+TTp{t))z{t) 



,1' t=l 

V iJi 



H N 



-Y.^s{t)\Y^Y.^[Df\t)]-P{t)\ (17a) 

t=l \h=l i=l I 

/ H N \ + 

s.t. z{t) = E E ^^^^t^ (^)] - ^(^) (17^) 

\/i=l z=l / 

constraints in (14b), (14c), (14d). (17c) 

Assuming the usual case of tTs [t) + TTp (<) > 0, we can further 
rewrite (17) as 

T 

min V(7r,(t) +7rp(i))z(t) 



,l'"i,2^--- t=l 

V i,h 



H N 



-E'^^W EEiI^[A'"W]-^W (18a) 

t=l i=l / 

if N 

s.t. z(t) > E E ^[^^''^ ~ ^ (^^^^ 

/l=l 1=1 

> V<, (18c) 
constraints in (14b), (14c), (14d). (18d) 

The ingredient of our decentralized stochastic optimization 
method is to consider the Lagrange dual [29] of the stochastic 
problem (18), analogous to the methods adopted in [30], [31] 
for other applications. Let X{t) > and ri{t) > be the dual 
variables associated with each of the constraints in (18b) and 
(18c). The dual function of (18) can be shown to be 

H 

J2 if Mt) + n^it) - X{t) = Tj{t) Vt, 

h=l 

— oo, elsewhere, 
where A= [A(l), . . . , A(T)]^ and 



min Ve 

(h) (fi) ^ — ' 

S.t. constraints in (14b), (14c), (14d). 
The dual problem of (18) is thus given by 

H 

max V$('')(A) 
A(t), ^ 

t=l,...,T 



h=l 



S.t. \{t) > V t = 1,...,T, 
>0, Vt = l,...,r, 



(19) 



(20a) 

(20b) 
(20c) 



One can see from the above equation that ?/(t) is in fact 
a dummy variable since it does not appear in the objective 
function. By combining (20b) to (20d), we then obtain 



H 



max y^C'^fA) 
A(t), ^ 

t=l,...,T '' = 1 

S.t. < \{t) < TTsit) + TTp{t) V i = l,...,r. (21) 

The dual optimization method for (18) is to iteratively solve 
the inner minimization problems in (19) and the outer maxi- 
mization part in (21) [27]. 

Distributed inner primal minimization: Let \{t;n) de- 
note the dual variable A(t) obtained at iteration n. Given 
X{t;n),t = l,...,r, the algorithm solves the inner mini- 
mization problems in (19) for all h = 1,...,H. Note that 
the objective function of (21) is a summation of $('*'' (A), h = 
1, . . . ,H, which is decomposable. Thus the inner minimization 
step can be carried out in a fully distributed fashion where 
each residence h solves the corresponding subproblem (19) 
independently. 

It is important to observe that subproblem (19) has exactly 
the same formulation as the selfish HEM design problem (6), 
except that, in (19), the hth residence is given a "pseudo price" 
X{t;n) ~ TTsit),t = 1,...,T. Therefore, the MDP method 
presented in Section 111-A can be directly used to efficiently 
solve (19) and obtain the optimal appliance control policy 
of (19) for each residence h. Let us denote {fi'f'^ {■■,n)}J^i, 
i = 1,...,N, (see (11)) as the optimal appliance control 
policy for (19) obtained by residence h at iteration n. More- 
over, let E{D[^\t; n)}, i = 1, . . . , N, t = 1, . . . ,T, he the 
corresponding expected scheduled loads for residence h. 

Distributed dual subgradient update: The dual variable 
A(t), t — l,...,r, can be updated by the subgradient 
projection method [32]: 

A(t; n + 1) =r \\{t; n) + c„ ( ^ ^ E[i5f' (t; n)] - P{t)] 



^h=l i=l 



Vt = 1, 



T 



where c,i > is the step size, and V{-) denotes the operation 
of projection onto the set [0,7rp(t) + 7Ts{t)]. 

In view of the fact that updating (22) requires the aggregate 
load X^ftLi Si^i ^[^i'''' ^)] °f all residences, it usually 
requires a control center to coordinate the residences for 
the dual update. To perform the dual update (22) in a fully 
decentralized fashion, we alternatively employ the consensus- 
subgradient method [33]. In this method, each residence h 
maintains a set of local copies of \{t;n), t = 1,...,T, 
denoted by A(''''(t;n), t = 1, . . . , T on its own, and locally 
updates them according to the subgradient of $(''-'(A): 



N 



c„ E[Z?f ) {t; n)] + E[U^'^^ {t)] - ^ (23) 

for t ~ 1, . . . ,T. Note that, in (23), only the local information 
about residence h is used, in addition to the bulk power 
purchase P{t) broadcasted by the retailer Since the desired 



TTpit) + TTsit) - Xit) = 7?(t) Vt = 1, . . . , T. (20d) 
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subgradient update (22) is the average of (23) over all h ~ 
1, . . . ,H, the second step in the consensus-subgradient method 
[33] is that each residence h exchanges with its connecting 
neighbors about u^''^'> {t; n) so as to achieve a consensus on 
A(t; n + 1). More precisely, residence h obtains A''*-' (i; n + 1) 
by 

A('')(t;n + 1) =r[f^{v'^^Ht;n),j e {^UMO} (24) 

for t — 1 , . . . , T, where Nh denotes the index set of the 
neighbors that can communicate with residence h, /(•) is an 
averaging consensus function (e.g., [33] 

ie{/i}uA^fe 

where W is an iJ by i7 mixing matrix), and the superscript 
indicates that the averaging consensus step is repeated for 
■0 times. 

The convergence properties of the consensus-subgradient 
method has been studied in [33]. Roughly speaking, the dual 
iterates X^'^^t;n + 1) for all h asymptotically converge to 
each other with a discrepancy no larger than ei > 0, and the 
corresponding dual objective value in (21) also asymptotically 
approaches the optimal value with a deviation no more than 
e2 > 0, where both ei and €2 are small positive numbers and 
decrease with ijj. This implies that, if the number of consensus 
steps ip is sufficiently large, the consensus-subgradient method 
converges to the optimal solution of (21). In computer simu- 
lations, we find that a small number of ip (e.g., ip < 15) is 
sufficient for achieving good convergence performance. 

Monte Carlo method for estimating J2Zi E{£'f ^ {t; n)}: 
In order to perform the dual update (23), each resi- 
dence h has to compute the expected scheduled load 
E{L'f ' {t;n)},t = l,..., T, associated with the control 
policy i = 1,...,VV. While it is difficult to 

obtain this expected power load analytically, one can estimate 
it through the Monte Carlo method [23]. In particular, the 
HEM unit in residence h can repeatedly generate realiza- 
tions of appliance requests t'l^}, ^1'^', ...(according to the 
customer's usage probabilities p\'^\t) and 7|^(i)), followed 
by applying the optimal control policy {fi[^} {■■,n)}f^^, i = 
1,...,N, at iteration ri; this outputs a simulated scheduled 
load profile E^i^F'^W' * = l,...,r. The HEM unit 
repeats this simulation multiple times, each of which outputs 
a scheduled load profile J2f=i ^I'^Ht), t = 1,...,T. By 
averaging J2iLi ^T^^i^)^ t ~ 1, . . . ,T, over all the simulated 
realizations, residence h can use this sample average as an 
estimate of Y,f^^ E{L>f ' (t; n)}, t = 1, . . . , T. Note that the 
computations mentioned above all can be implemented for 
each appliance and for each realization in a parallel manner. 

We summarize the decentralized stochastic optimization 
algorithm in Algorithm 1 . Three remarks regarding Algorithm 
1 are in order: 

Remark 2 We should emphasize that the above optimization 
method for (21) can be done in an off-line fashion since solving 
(21) uses only the statistical information of customers (i.e., 
-Pi''^(Oi 7i m(^))- ^ result, we only need to perform the 



Algorithm 1 Decentralized stochastic optimization algorithm 
for (14) 

1: Input an initial set of A^^^ {t;0),t = 1, . . . , T, at residence 

h, for all h^l,...,H. 
2: Set n = 0. 
3: repeat 

4: tor h = l, ..,H do 

5: 1) Given {A('''(i; n) — Trs{t)}J^i, residence h solves 
(19) by the MDP method in Section III- A to 
obtain the optimal control policy of appliances 

6: 2) Residence h applies the instantaneous con- 
trol policy {ii'i''^{-;n)}f^i,i ~ 1,...,N, and the 
Monte Carlo method to estimate the expected load 
j:l,E{Dl'^\t;n)},t^l,...,T. 

1: 3) Residence h obtains v^^'^ {t; n) by (23). 

8: 4) Residence h exchanges (t; n) with its connect- 
ing neighbors for updating \'-'^\t; n + 1) by (24). 

9: end for 
10: n = n + 1 

11: untU the predefined stopping criterion is satisfied. 
12: All residences respectively apply the running-averaged 
control polices in (25) for on-line (real-time) scheduling. 



decentralized optimization once for each look-ahead horizon 
T. The associated optimal control policy of appliances can 
then be applied to the real-time scheduling process. Note that 
this is very different from the model predictive CEC method 
in [28] where distributed optimization has to be carried out 
T — 1 times. Therefore, the proposed Algorithm 1 has a much 
reduced computation and communication overheads for the 
CoHEM network. We should also mention that Algorithm 
1 works well under the assumption that the retailer has 
reasonably accurate estimates of the real-time balancing prices 
and {^p(0}?:i. 

Remark 3 Since the proposed approach is based on 
the dual optimization (21), the primal control policy 
{fiff{-;n)}J^i,i = l,...,N, obtained at each iteration n, 
may not converge as well as the dual variables {A^''-'(t; n)}f^i. 
In that case, following the same spirit as in [34], [35], one can 
alternatively use the running averaged policy 

round I i ^ ^f;} (X; ^) I e {0, 1} VX 6 A-f ^ , t, 

(25) 

(see Step 12 of Algorithm 1), where the operator round(-) 
rounds the averaged policy to its feasible region. We find 
through simulations that this running-averaged policy works 
well in practice. 

Remark 4 The proposed method is suboptimal compared to 
the original CoHEM design problem (9) since it is optimizing 
the lower-bound problem in (18), and (18) and its Lagrange 
dual in (21) have a non-zero duality gap in general [30], 
[31]. The suboptimiality can be measured as follows. Let us 
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denote the optimal objective value of (9) by /*, denote that 
of the lower-bound problem (18) by /;*, and denote that of 
the dual problem (21) by f^^. Then we have /* > > f^^ 
by weak duality. Suppose that the running-averaged control 
policy in (25) corresponds to an empirical primal objective 
value fp{n) > for (9) at iteration n, and the averaged 
dual iterates '^}^^i^^^\t\n)/ H, t = l,...,r, correspond 
to an empirical dual value fid{n) < f*^ for (21). The 
normalized approximation gap between fp{n) and /* can be 
upper bounded as 



J V 



fid{n) 



Vn, 



(26) 



where the right hand side term is the empirical duality gap 
which can be evaluated numerically and will be examined in 
Section IV-D. 



By following the same reformulation steps and dual decompo- 
sition technique in (17) to (21), we can obtain the dual problem 
of (28) as 



H 



max 



t=l,...,T 



h=l 



S.t. < \{t) < TTsit) + TTpit) Vt = 1, . . . , T, (29) 

where $('')(A), similar to (21), is given by 



$('')(A) 



N 



s.t. constraints in (14b) to (14d), 



(30) 



C. Extensions 

In this subsection, we discuss two interesting extensions of 
the proposed CoHEM design. 

1) Joint power procurement and CoHEM optimization: 

As we mentioned in Remark 2, the proposed approach opti- 
mizes the CoHEM scheduling in an off-line manner Therefore, 
it is possible to optimize the CoHEM scheduling for the next 
day and determine the power bid {-P(<)}^i jointly^, provided 
that the aggregator can accurately estimate the locational 
marginal price (LMP) in the day-ahead wholesale market and 
the real-time prices {Trs{t)}f^i and {'^p{t)}f^i beforehand. 
Note that the two problems have different time scales - the 
power procurement is in the day-ahead market where the 
retailer submits bids B{1), . . . , _B(24), for 24 hours of the next 
day; while the CoHEM scheduling is for the real-time market 
where the unit of (discrete) time is usually in minutes. For ease 
of illustration, let us assume that sampling time interval in real 
time is 15 minutes. Then P{t) = B( [i/4] ) for t = 1, . . . , 96. 
Let Ch,e{B{£)) denote the (convex) cost for the power bid at 
hour £ = 1, . . . , 24. By adding Chj{B{^)) to the real-time 
cost in (8), the total cost of the retailer is given by 



Cost = 



nAt)iB{\t/4])~Y.^[L['oLM 



(27) 

and the associated joint power procurement and CoHEM 
scheduling problem can be formulated as 

min Cost function in (27) 

B(f)>0,£=l,....24. 

S.t. (9b) and constraints in (14b) to (14d). (28) 

'if the hour-ahead market is available, then the retailer can also jointly 
determine the CoHEM scheduling and the power bid for the next hour. Here, 
we illustrate the joint design problem by assuming the day-ahead market only. 



and 5* (A) is given by 
*(A) = 



96 



°r J2^^Am) - E(^w - ^sit))Bi\t/4\). (31) 



One can see from (29) that the optimization of $^'')(A), 
h ~ 1, . . . , H, and 4* (A) are completely separable, and thus, 
the inner primal minimization of (29) can be carried out in a 
fully parallel manner In particular, (30) can be solved by the 
MDP method in Section III- A, while (31) is a convex problem 
which can be solved by off-the-shelf convex solvers [36]. The 
only difference from (20) is that either an aggregator or one 
of the customers may need to be in charge of solving (31). 
Then the joint design problem (27) can be handled in a similar 
decentralized fashion as Algorithm 1 . Simulation results to be 
presented in the next section will show that this joint power 
procurement and CoHEM scheduling can further reduce the 
overall cost of the retailer 

2) Extension to general convex cost functions: In some 
cases, the real-time cost function can be more complicated 
than that in (8) [18] . The proposed CoHEM design problem 
(9) and Algorithm 1 can be extended to other general (convex, 
increasing) real-time cost functions. To illustrate this, let us 
rewrite (14) as follows 



T 



mm 

'^i.l '*i.2 >••• 4 = 1 

V i,h 



H N 



h=l i=l 
H N 



\h=l i=l 

S.t. constraints in (14b) to (14d), 



(32) 



where Cs_t[-] and Cp_t[-] denote the cost functions for buy- 
ing additional power and absorbing extra power at time t, 
respectively; they are assumed to be convex and increasing. 
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By introducing the two slack variables 



/ H N \ + 

\h=l i=l / 
/ H N \ + 

yW= M'W-EEiEI^F'^W] vt, (34) 

V h=l i=l I 



one can write (32) as 

T 



^^min Ej^Ml^Wl + C^P.tbW] 



(35a) 



V i.h 



s.t. constraints in (14b) to (14d), 

z(t) > 0, y{t) > 0, yt, 



H N 



^W^EEi^[A'"W]-^'W Vt, (35b) 

h=l 4=1 

if JV 

2/W>i^W-EElI^[A''^W]^*- (35c) 

h=l i=l 

Similar to (20), one can show that the dual problem of (35) 
has a separable inner minimization part, but has two additional 
minimization subproblems. These two subproblems can be 
handled by an aggregator or one of the voluntary residences. 
Moreover, the dual variables associated with constraints (35b) 
and (35b) can be updated by the consensus projected subgradi- 
ent method, analogous to Algorithm 1 . The detailed derivations 
are omitted here. 

IV. Simulation Results 

Extensive simulation results are presented in this section to 
examine the performance of the proposed CoHEM architecture 
and Algorithm 1. 

A. Simulation Setting 

We consider a scenario where there are H residential 
units, with 4 deferrable appliances in each residence. The 
optimization horizon is set to 96 (T = 96) which corresponds 
to a whole day with 24 hours and 4 quarters for each hour. 
The four controllable appliances considered are respectively 
washing machine, dish washer, tumble dryer and PHEV, all 
assumed to have a single operation mode and a single power 
profile for simplicity. The load profiles of the first three 
appliances are obtained according to the discrete load models 
in [25, Table Bl]; the load profile of PHEV is set to be 
constant when on, with an instantaneous power consumption 
of 3 kW, and a working duration uniformly generated between 
1 to 6 hours. We follow the synthetic method proposed in 
[25] to generate the request probabilities p\'^\t) for the first 
three appliances in each house. For the PHEV, the request 
probability is set to 0.8 for three times that are uniformly 
distributed between 8 am and 12 pm, 5 pm and am, and 
am and 2 am, respectively. The deadline constraints for the 
washing machine and dish washer are uniformly generated 
between 15 minutes to 2 hours, and for the tumble dryer and 
PHEV, they are generated uniformly between 15 minutes to 3 
hours. The uncontrollable load U'-'^\t) is contributed by the 



other 14 appliances listed in [25, Table Bl] and is generated 
following the synthetic method in [25]. We assumed that each 
residence h can accurately estimate (t), for all h. 

If not mentioned specifically, the setting of Algorithm 1 is 
as follows. The initial value A^'*' {t: 0) is set to — 



^ for 

all t = 1, . . . , 96, and /i = I, . . . ,H; the step size c„ is set to 
5/(n + 5); the number of averaging consensus step ip is set to 
15; 100 randomly generated realizations are used to estimate 
E!Ii IE{-DfV; n)} in the Monte Carlo method. Algorithm 1 
is run for a predetermined number of iterations equal to 200. 

The model predictive CEC method in [28, Algorithm 1 & 
Algorithm 2] is also simulated. The initial values, step size, 
and number of averaging consensus steps of [28, Algorithm 
2] are set to the same values as Algorithm 1. The maximum 
number of iterations of [28, Algorithm 2] is set to 150. Note 
that, according to [28, Algorithm 1], [28, Algorithm 2] has to 
be carried out 95 times, from time 1 to time 95. 

For ease of elaboration, we set both prices TTs{t) and 7rp(i) 
to one for all t = 1, . . . , 96. In this case, the real-time cost in 
(8) reduces to the total deviation between the aggregate load 
J2h=i ^totai(^) '^^^ '■^^ day-ahead power purchase {P{t)}f^i 



H 

Deviation Cost = E ^(^) " E ^lotai(0 

h=l 



(36) 



The day-ahead bits {P{t)}J^^ are generated as follows. 
Given the usage probabilities of customers, we use the Monte 
Carlo method to generate 50 realizations of aggregate unsched- 



-(h) 



uled deferrable loads X^ftLi X^iLi -^i 
(1)), by which an estimate of X]^=i X]t=i '^[^\ 



{t),t 



- 1, . . . , 96 (see 
'^^^(t)\, denoted 
by L(t), is obtained by taking the sample average. The day- 
ahead bid {-P(t)}^i used in the simulations is obtained by 



1 



16 



^(^) = Y%^ ^^^^^ - 1) + + ^ 

i=\ h=l 



for t ~ 1, . . . , 96. Note that in the first term of the above 
equation, we applied a peice-wise averaging for every 16- 
sample interval in order to emulate the effects of generation 
ramping constraints and that the retailer may have imperfect 
statistical information of customers and renewables^. 

B. Selfish HEM v.s. CoHEM 

Since there is no existing work that considers the same 
network utility and probabilistic models in this paper, we 
focus on comparing the proposed CoHEM architecture and 
Algorithm 1 with the selfish HEM system (6) and the model 
predictive CEC method in [28] . Figure 3 shows the simulation 
results of the unscheduled load and the load scheduled by 
selfish HEM systems, in the presence of 100 residential units 
{H = 100). All the presented results are averaged over 50 
randomly generated request realizations. In the simulation of 
selfish HEM, the residences are given a price signal that 
is inversely proportional to the power bid of the grid, i.e., 
TT{t) = 1/P{t), in order to motivate the HEM systems in 

*The way we generate the day-ahead bid may not always be satisfactory 
from a practical point of view; however it is sufficient for one to assess and 
compare the scheduling capabiHties of the developed DR algorithms. 
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Fig. 3: Simulation results of the unscheduled power load 
and the power load scheduled by selfish HEM systems. The 
number of residences H is set to 100. 

the residences to move their loads to the high-supply period. 
One can see from Figure 3 that the selfish HEM design 
successfully moves the load to the high-supply region, but 
that also causes significant rebound peaks. The aggregate 
deviation in (36) corresponding to the unscheduled load is 
2823.3 kW, but the deviation corresponding to selfish HEM 
increases to 4350.4 kW. This result shows that the selfish HEM 
without coordination between the neighborhood would result 
in significant power imbalances and consequently considerable 
real-time cost to the retailer 

Figure 4(a) displays the power load scheduled by the model 
predictive CEC method in [28, Algorithm 1 & Algorithm 2]. 
By comparing Fig. 4(a) with Figure 3, one can observe that the 
scheduled aggregate load by the model predictive CEC method 
[28] can follow the power supply, and the corresponding 
deviation cost dramatically decreases to 1549.8 kW. Figure 
4(b) presents the power load scheduled by the proposed 
Algorithm 1. As one can see from this figure. Algorithm 1 
performs comparably as the model predictive CEC method in 
[28], but it has a slightly higher average deviation of 2002.7 
kW. Compared to the deviations of the unscheduled loads 
(2823.3 kW) and the selfish HEM system (4350.4 kW) shown 
in Fig. 3, Algorithm 1 yields around 29% and 53% reductions, 
respectively. We should emphasize again that Algorithm 1 
has a much lower communication overhead than the model 
predictive CEC method in [28]. In particular, according to 
the setting described in Section IV-A, Algorithm 1 requires 
a total of 200 x 15 = 3000 message exchanges; whereas 
the model predictive CEC method in [28] requires at most 
150 X 15 X 95 = 213750 message exchanges. Figure 4(c) 
further shows the power loads scheduled by Algorithm 1 and 
the selfish HEM system in a neighborhood of 400 customers 
(H = 400). Again, we see that the proposed CoHEM archi- 
tecture and Algorithm 1 can significantly improve the power 
balancing and real-time cost of the aggregator 

To further look into how the number of residences affects 
the performance of the proposed CoHEM architecture, we list 
in Table I the normalized average deviation cost for different 
numbers of residences. The results are obtained by testing 
the associated scheduling policy output by Algorithm 1 over 
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(a) Model predictive CEC metliod in [28] (H = 100) 
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Fig. 4: Simulation results of the loads scheduled by the model 
predictive CEC method in [28] and Algorithm 1. 
100 randomly generated request realizations. We see from 
Table I that there is a significant drop of the normalized 
deviation cost from 54.5 kW to 22.4 kW when the number of 
residences increase from 5 to 50; after H > 50, the normalized 
deviation costs remain relatively constant, showing that the 
performance of Algorithm 1 is quite robust against the size of 
the neighborhood. 

C. Robustness of CoHEM 

While all the residences in the neighborhood should coop- 
eratively participate in the proposed CoHEM program, it is 
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TABLE I: Normalized average deviation cost (in kW) versus 
number of residences. 



TABLE IL Average computation time (in seconds) of Algo- 
rithm 1 versus number of residences. 



H 
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Deviation cost/_ff 


54.5 


46.4 


22.4 


21.4 


22.9 


20.6 



The results are obtained by averaging over 100 simulation realizations. 
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Fig. 5: Simulation results of deviation cost of Algorithm 1 
in the presence of non-cooperative customers; the simulation 
setting is the same as that for Fig. 4(b). 

possible that there are some non-cooperative residences who 
selfishly keep using the selfish HEM policy. Here we examine 
how the number of such non-cooperative residences affect the 
performance of the proposed CoHEM architecture. Figure 5 
presents the deviation cost in (36) for different number of non- 
cooperative residences. The simulation setting is the same as 
Fig. 4(b). We can see from this figure that the deviation cost of 
the aggregator increases when there are more non-cooperative 
residences. However, compared to that without scheduling (the 
green dashed line), we observe that the aggregator can still 
make a profile out of the CoHEM program as long as there are 
more than 50% of residences in the neighborhood willing to 
follow the CoHEM scheduling policy. This result demonstrates 
the robustness of the proposed CoHEM design against non- 
cooperative residences. 

D. Convergence and Complexity 

In this subsection, we examine the convergence behavior of 
Algorithm 1 (see Remark 4) and its computation times. Figure 
6(a) displays the empirical objective value fid{n) of the dual 
problem (21) versus the iteration number, for various numbers 
of residences in the neighborhood. Figure 6(b) shows the 
corresponding normalized duality gap as discussed in (26). We 
can see from these figures that, within 150 iterations, the dual 
updates of Algorithm 1 as well as the normalized duality gap 
converge asymptotically. The normalized duality gap shown in 
Figure 6(b) shows that the gap between the empirical objective 
value fp{n) of the original problem (9) and fid{n) of the dual 
problem (21) could be large; however, this does not necessarily 
imply that the CoHEM scheduling solution obtained from 
Algorithm 1 is far away from the optimal solution of problem 
(9), i.e., the gap between fp{n) and /* is not necessarily large. 
To further examine this aspect, we conduct a simulation where 
we set = 2 and = 1 (two residences and each of 



H 


50 


90 


170 


210 


400 


Tc/itc 


79.367 


126.258 


272.612 


330.771 


562.284 


n/itc/H 


1.587 


1.402 


1.603 


1.575 


1.407 



* Tc/itc stands for the average computation time per iteration and Tc/itc/H 
represents the computation time per iteration and per residential unit. 



the residences has only one appliance). Under this setting, 
we are able to apply the MDP technique (as discussed in 
Section III-A) to exhaustively find the optimal control policy 
for the CoHEM problem (9) and the corresponding optimal 
objective value. In Fig. 6(c), we plot fp{n) and fid{n) of 
Algorithm 1 and also the optimal value of /* obtained from 
the exhaustive MDP search. Specifically, at iteration 200, we 
have /p(200) 161.4 and fid{200) = 116.7 while the optimal 
objective value is /* = 156.7. One can see from this figure 
that, although there is a large gap between fp{n) and fid{n), 
fp{n) is actually close to /* (with a normalized accuracy 
0.029). While such inspiring result may not always hold true 
for large-scale problems (i.e., when H and N are large), the 
evidenced results in Figures 3, 4 and 5 have demonstrated that 
Algorithm 1 is practically effective for large scale scenarios 
and can yield promising performance improvement for real- 
time power balancing. 

Table II lists the computation times (in seconds) of Algo- 
rithm 1 for various numbers of residences in the neighborhood. 
The algorithm was run on the Matlab platform using a com- 
puter with a 4-core 2.6 GHz CPU and 12 GB RAM. Note 
that while Algorithm 1 is a decentralized algorithm and the 
computations involved for solving (19) and the Monte Carlo 
method can be parallelized, they can only be implemented 
sequentially in a computer The first row of Table II shows 
the average computation times per iteration (averaged over 400 
iterations) (Tc/ite) and the second row shows the computation 
times per iteration and per number of residences (Tc/itc/H). It 
is interesting to see that Tc/itc/H remains relatively constant 
when H increases, demonstrating that Algorithm 1 is truly 
scalable with the number of residences as long as a parallel 
computation can be implemented. 
E. Joint power procurement and CoHEM scheduling 

In this subsection, we examine the performance of the 
joint power procurement and CoHEM scheduling formulation 
discussed in Section III-C. We consider the real-time deviation 
cost in (36) and the following quadratic cost function for power 
procurement 

C^,AB{^)) = W^LMP W, ^ = 1, . . . , 24, 
where ttlmpI^) denotes the LMP for hour I. In the simula- 
tion, the LMP {'^'LM{f)}fLi obtained from https://www2. 
ameren.com/RetailEnergy/realtimeprices.aspx, on day June 21, 
2012. We put more weights on mitigating the power imbalance 
by considering the following weighed cost 

96 H 24 
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Fig. 6: Convergence curves of Algorithm 1. 

As a comparison with the joint power procurement and 
CoHEM scheduUng design, we also simulate a scenario 
where the retailer first obtains a power bid {B{£)}j^^ sep- 
arately, followed by performing CoHEM scheduling (Algo- 
rithm 1) based on this power bid. In particular, the power 
bid is predetermined by minimizing the cost in (37) with 
J2h=i^[-^totaii^)^ replaced by the aggregate (unscheduled) 



power load J2h=iJ2iLi'^[Li'^\t)] (which can be estimated 
through the Monte Carlo method. The uncontrollable loads 
U^'^\t) are neglected here). The obtained power bid and 
the corresponding CoHEM scheduled load (averaged over 
50 appliance request realizations) are presented in Fig. 7(a), 
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Fig. 7: Performance of joint power procurement and CoHEM 
scheduling for a scenario with 130 residences {H = 130). 

under the same simulation setting as that in Fig. 3. The 
associated cost for power procurement X^fc^ -^^(^)^lm(^) 
and the average real-time deviation cost IEEt=i [^(TV^l) ^ 
J2h=i -^totai(OI] given by 1685.0 and 1874.4, respectively, 
leading to a total cost of 3559.4. The jointly optimized 
power bid and the CoHEM scheduled load are shown in 
Fig. 7(b). The corresponding cost for power procurement 
Y^'i'ti W'^Lui^) and the average real-time deviation cost 

EEt!i \m/^^)-Eh=l ^SaiWI] are given by 1111.0 and 
1427 A, respectively, which results in a lower total cost of 
2538.8 (28.6% reduction compared to 3559.4 in Fig. 7(a)). We 
see that the joint design can yield lower costs for both real- 
time deviation and power procurement. We should emphasize 
here that the joint power procurement and CoHEM scheduling 
design requires accurate estimates for the LMP {7rLM(^)}fii 
in the day-ahead market as well as for the real-time prices 
{ns{t)}'t=i and {7rp(i)}?Li for the next whole day. Further 
investigations taking into account possible price estimation 
errors are needed in the future. 

V. Conclusions and Future Directions 

In the paper, we have presented a CoHEM architecture that 
coordinates the home energy scheduling of multiple residences 
in order to reduce the real-time power balancing cost. We first 
proposed a simple MDP approach for modeling the deferrable 
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appliances and solving the individual HEM design problem. 
Then, we presented a decentralized algorithm (Algorithm 1) 
for handling the CoHEM design problem. The presented sim- 
ulation results have demonstrated that the proposed CoHEM 
design as well as its decentralized algorithm can effectively 
decrease the real-time power balancing cost of the retailer. 

In the future, we will extend the proposed load model 
and decentralized algorithms to thermostatically controlled 
appliances (e.g., heating, ventilating and air conditioning 
(HVAC) [9]). In particular, since HVAC has much shorter 
duty cycles compared to the non-interruptible loads, it can be 
scheduled myopically to reduce the uncertainty on P{t) due 
to imperfect information about the renewable energy sources 
and uncontrollable loads. In addition, it would be interesting 
to integrate the CoHEM with storage device control as well as 
distributed power generation control [37], and study this joint 
power flow control and CoHEM design problem form both 
economic and algorithmic aspects. 
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